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ABSTRACT 

In 1989 the Department of Education contracted with 
Development Associates, Inc., to conduct a review of the evaluation 
and improvement practices of projects funded under Title VII of the 
Elementary and Secondary Education Act, often known as the Bilingual 
Education Act. The study focused on local transitional bilingual 
education and special alternative projects and was designed to 
describe and assess local evaluation practices and the use of 
evaluation results. This paper summarizes somui of the highlights of 
the findings and conclusions of the study. Three major activities 
were conducted: (1) a file review of the applications and evaluation 
reports of a stratified random sample of 200 Title VII projects 
funded in fiscal year 1989; (2) a mail survey of all project 
directors and evaluators of 655 projects funded in that year; and (3) 
case studies of evaluation systems of 18 local projects. Findings 
indicate that the purposes and audiences for Title VII evaluations 
have not been clearly articulated by the U.S. Department of 
Education, and that there has also been no clear description of how 
the evaluation needs of the department and local proj^jcts should be 
integrated or prioritized. It was also found that evaluation reports 
are integrated or prioritized has not been made clear. Evaluation 
reports are not being used systematically, and, in general, their 
quality is only poor to adequate. It is concluded that there is an 
inherent tension between evaluation that is primarily conducted for a 
federal sponsor and that which is conducted for a local project. Two 
tables summarize evaluation purposes and needs. (SLD) 
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In 1989, the U.S. Department of Education contracted with Development Associates, 
Inc., to conduct a review of the evaluation and improvement practices of projects 
funded under Title VII of the Elementary and Secondary Education Act, which is 
often known as the Bilingual Education Act. The study focused on local Transitional 
Bilingual Education and Special Alternative projects, and was designed to describe 
and assess local evaluation practices and the use of evaluation results. This 
presentation summarizes some of tlie highlights of the findings and conclusions of 
the study, and raises some issues which are relev?nt to many other programs with 
requirements for outside evaluations. 



II. Methodology 

The study consisted of three major activities: (1) a file review of the applications and 
evaluation reports of a stratified random sample of 200 Title VII projects funded in 
FY 1989; (2) a mail survey of all project directors and evaluators of all 655 projects 
funded in that same fiscal year; and (3) case studies of the evaluation systems of 18 
local projects. In addition, we conducted interviews with all of the federal Project 
Officers of local Title VII grants within the Office of Bilingual Education and Minority 
Languages Affairs (OBEMLA), and with other selected education officials at the state 
and local levels. 
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IIL The Title Vn Program Evaluation System 



Through Title VII school districts compete for grants to support demonstration and 
capacity building projects to serve limited English proficient students. Title VII 
grants are for three years, with a virtually automatic two year extension available for 
the asking. 

Since 1968, there have been requirements for Title VII projects to conduct evaluations 
of their activities. The structure and content of those evaluations have been guided 
by a number of requirements and resources defined and offered by the Department 
of Education. The evaluation requirements for Title VII, Part A projects are defined 
in Section 7033 of the Bilingual Education Act. Those requirements are expanded 
into regulations in Sections 500.50, 500.51, and 500.52 of the federal code (34 CFR). 
They include the following: 

Section 500.50 

A grantee's evaluation design must include a measure of the educational progress of 
project participants when measured against an appropriate nonproject comparison 
group. 

The evaluation results must be computed so that the conclusions apply to the 
persons, schools, or agencies served by the projects. 

The evaluation instruments used must consistently and accurately measure progress 
toward accomplishing the objectives of the projects, and they must be appropriate, 
considering factors such as the age, grade, language, degree of language fluency and 
backgrotmd of the persons served by the project. 

T\ 3 evaluation procedures must minimize error by providing for proper 
administration of the evaluation instruments, at twelve-month testing intervals, 
accurate scoring and transcription of results, and the use of analysis and reporting 
procedures that are appropriate for the data obtained from the evaluation. 

The evaluation procedures must provide objective measures of the academic 
achievement of participants related to English language proficiency, native or second 
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language proficiency (for programs of development bilingual education), and other 
subject matter areas. 

A grantee's evaluation must provide information on the academic achievement of: 

(A) Current participants in the project, who are: 

1. Children who are limited English proficient; and 

2. Children whose language is English; and 

(B) Qiildren who were formerly served in the project as limited English 
proficient, have exited from the program, and are now in English 
language classrooms. 

This information must include: 

(A) The amoimt of time (in years or school months, as appropriate) the 
participants received instructional services in the project and, as 
appropriate, in another instructional setting; 

(B) The participants' progress in achieving English language proficiency 
and, for programs of developmental bilingual education, progress in 
another language; and 

(C) The former participants' academic progress in English language 
classrooms. 

Section 500.51 

In carryirig out the annual evaluation under S500.50, a grantee shall collect 
information on: 

(A) The educational background, needs, and competencies of the limited 
English proficient persons served by the project; 

(B) The specific educational activities tmdertaken pursuant to the project; 

(C) The pedagogical materials, methods, and techniques utilized in the 
program; 

(D) With respect to classroom activities, the relative amounts of 
instructional time spent with students on specific tasks; 

(E) The educational and professional qualifications, including language 
competencies, of the staff responsible for plaiming and operating the 
project; and 

(F) The specific activities tmdertaken to improve prereferral evaluation 
procedxires and instructional programs for LEP children who may be 
handicapped or gifted and talented. 
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Section 50052 

A grantee shall report to the Secretary annually, the information collected in S500.51 
and an evaluation of the overall progress of the project including the extent of 
educational progress achieved through the project measures, as appropriate, by: 

(A) Tests of academic achievement in English language arts and, for 
programs of developmental bilingual education, second language arts; 

(B) Tests of academic achievement in subject matter areas; and 

(C) Changes in the rate of student: 

(1) grade-retention; 

(2) dropout; 

(3) absenteeism; 

(4) placement in programs for the gifted and talented; and 

(5) enrollment in postsecondary education institutions. 

Section 501.30 of the regulations describes the basis on which applications for Title 
VII grants are evaluated/ The evaluation plan is worth 8 out of 100 points. 
According to the regulations, "The Secretary reviews the strength of tlie evaluation 
plan and its relationship to the educational goals of the project and the activities 
conducted to attain those goals." 



In addition to the Federal statute and regulations, the Title VII evaluation system 
includes guidance materials developed by the U.S. Department of Education for local 
projects plus two Evaluation Assistance Centers. 

The Bilingual Education Evaluation System (BEES) was published in November of 
1987. It consisted of a User's Guide in two volumes (Volume I, Recommended 
Procedtires; Volume II, Technical Appendices), plus a brief document entitled 
"Abbreviated Recommendations for Meeting Title VII Evaluation Requirements." 

The BEES is a total evaluation system that involves a process evaluation component, 
an outcome evaluation component, end procedures for integrating the two. The most 
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innovative element of the system was the gap-reduction design that was 
recommended for assessing student outcomes. 

At approximately the same time that the Department contracted for the development 
of BEES it also funded two regional Evaluation Assistance Centers (EACs). These 
centers were designed to assist Title VII grantees and other providers of services to 
LEP students with materials, training, and technical assistance related to evaluation. 
Such assistance was previously provided through Evaluation, Dissemination, and 
Assessment Centers (EDACs) and through Bilingual Education Multifunctional 
Support Centers (BEMSCs). These other Centers had multiple functions. The EAvJs 
were developed to focus specifically on evaluation. 

The EACs have assisted OBEMLA by providing presentations at Management 
Institutes for new projects, they have prepared various materials for projects, and 
they have provided site-specific assistance through mail and telephone consultation, 
regional training events, and on-site technical assistance. 

IV. Purposes of the Current System 

Evaluations of Title VII projects can serve a variety of purposes at the federal, state, 
and local levels. There is no explicit statement of purpose in Title VII legislation or 
regulation, and in discussions with OBEMLA and other Department staff and 
officials, there was no consensus concerning evaluation purposes. In thinking about 
Title VII evaluations, we have identified nine major potential purposes which such 
evaluations could serve. We divide those purposes into three major categories: (a) 
those that primarily address federal needs; (2) those that primarily address local 
needs; and (3) those that address both federal and local needs. In describing these 
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purposes, we make a distinction between the national Title VII "program" and the 
specific "projects" which are implemented by local school districts. 

In Table 1 these nine evaluation purposes are further described in terms of the key 
evaluation questions related to each. In general, outcome evaluations of projects 
focus on the sixth and ninth purposes, while process evaluations focus on the seventh 
and eighth purposes. 

The Title VII evaluation regulations as they are presently written . uggest a focus on 
the first four plus the eighth and ninth of these purposes. The regulations explicitly 
refer to information which is to be provided to the Secretary of Education, and the 
contents of the required reports suggest the types of uses which could be made. In 
communications with projects (through Management Training Institute sessions on 
evaluation, distribution of BEES, and EAC assistance), it also appears that OBEMLA 
promotes the sixth and seventh purposes on the list (project effectiveness assessment 
and project and program improvement). However, there is no clear statement of 
purpose for Title VII evaluations in the legislation or regulations, and OBEMLA does 
not clearly indicate how evaluation data are used by federal audiences. Thus, local 
school officials and local project staff are imclear about why and how evaluation data 
should be collected and used. 
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TABLE 1 

Title VII Evaluation Purposes and Related Evaluation Questions 



Evaluation Purpose 


Evaluation Questions 


Federal 

1. Compliance 
monitoring 


1. Are the intended people being served? 

2. Are funds being ppent appropriately? 

3. Are the program goals being addressed? 


2. Grant funding 
evaluation 


1. Is the project being implemented as proposed? 
z. L/oes me quauvy or project inipiemenianon jusnry 
refunding? 


effectiveness 
assessment 


1 Tc fViPrP HpmnT\QH*aHlp nrnOTPQ^ fnxvarH rpapHitity^ nrnoTflTH 

goals? 

2. Is the progress toward program goals v^orth the costs 
(including opportunity costs)? 

3. Are the program goals still relevant to the needs? 


4 Meta-analvtic 
research 


1. What approaches work best with specific populations? 

2. What approaches have been shown not to be effective? 


5. Exemplaiy project 
identification 


1. Which projects demonstrate exemplary practices? 

2. Which projects or practices can be replicated elsewhere? 


Local 

6. Project 
effectiveness 
assessment 


1. How successful was the project in meeting its objectives? 

2. Which project components were more or less effective? 

3. What project components should be continued after 
federal funding ends? 


7. Project 
improvement 


1. Were specific activities implemented as planned? 

2. Were intended short-term results achieved? 

3. How can project outcomes be improved? 


Federal and Local 

8. Technical 
monitoring 


1. Do project activities involve accepted educational 
practices? 

2- Are the practices being used appropriate tor the 
populations being served? 


9. Project and 
program 
promotion 


1. Which project outcomes are most salient and important? 

2. To whom are those outcomes most important? 



AERA PRESENTATION 
AprU 1993 



M.Young & P. Hopstock 
Development Associates, Inc. 



10 



ERIC 



-9- 



V. Summary of Major F'jidings 

A. Purpofies and Uses of Evaluations 

1. As stated above, the purposes and audiences for Title VII evaluations have 
not been clearly articulated by the U.S. Department of Education. Indeed, 
there is disagreement within the Department concerning the purposes 
local project evaluations are to serve. 

2. There also has been no clear description of how die evaluation needs of 
the U.S. Department of Education and of local Title Vn projects should be 
integrated, or of how to prioritize them when they are in conflict. 

3. Of the possible federal uses of Title Vn evaluation reports, few, if any, are 
being realized under the present system. Reports are not being used 
systematically by the Department to assess Title VU at either the program 
or project level, and a majority of project directors believe that evaluations 
of projects could be more useful to those projects tlian they presently are. 

B. Quality and Cost of Title VII Evaluations 

1. In general, the quality of Title VII evaluation reports could be described 
as from "poor" to "adequate." There is considerable room for 
improvement in the evaluations which are done and the evaluation 
reports which are submitted. 

2. The average project reported collecting data on three-quarters of the data 
elements required by Title Vn regulations, and including data on only half 
of them in their 1989-90 evaluation report. 
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3* An issue frequently encountered by evaluators is the poor quality of data 
collection systems used by projects. The lack of available high quality 
process and outcome data hinders many evaluations. The extent to which 
the improvements of data collection/testing procedures stimulated by 
evaluation recommendations helped the projects in general, or just helped 
improve future project evaluations, could not be determined. 

4. Although the cumulative amount spent on Title VII evaluation is quite 
large, the amount spent for individual evaluations is small. It is 
unreasonable to expect a detailed and comprehensive evaluation of a 
major project for $3,000 to $4,000, and it is also unreasonable to expect a 
meaningful process evaluation when evaluators only spend 4-5 days on- 
site per year. 

C. Evaluation Implementation 

1. Because of the small amoimt of funds available for evaluation in most 
projects, project directors often ask evaluators to focus on outcome 
evaluation. There tends to be little mvolvement by evaluators in the 
evaluation of project implementation processes. 

2. Perhaps for this reason, evaluators are particularly involved in and 
interested in achievement test results, while project directors and project 
staff are particularly interested in recommendations for improving the 
design and operations of the project. If project staff had complete control 
over evaluation designs, there would be much less time and money spent 
on standardized testing of students. 

3. In most cases, those performing Title VII evaluations have appropriate 
qualifications. The a^-erage Title Vn project was evaluated by an 



AERA PRESENTATION 
April 1993 



12 



M. Young & P. Hopstock 
Development Associates, Inc. 



- 11 - 

inaividual with 15 years of experience in evaluating educational programs, 
and 10 years of experience ev?Juating Title VII projects. 



4. There is no convincing evidence indicating the superiority of either 
external evaluators or district evaluation staff. The ability and motivation 
of the evaluator are more important than that person's organizational 
affiliation. 

5. Evaluators who do multiple Title VII evaluations are more likely to meet 
the mandates of Title Vn evaluation regulations. However, they are not 
any more likely to meet the needs of local projects. 

VI. Conclusion 

In conclusion, we believe there is an inherent tension between evaluation which is 
primarily conducted for a federal sponsor and evaluation which is conducted for a local 
project. For the federal sponsor, monitoring and compliance reWew are important 
elements of the evaluation process, while for the local project, monitoring and compliance 
review may actually interfere with the coliegial relationships needed to carry out 
successfully process evaluation. Similarly, the reporting approaches which are most 
useful to local projects (quick, infonrxal, and focused on specific individuals or activities) 
have little utility for federal audiences. In Table 2 we rate the importance of the nine 
purposes of Title VII evaluations for three audiences: federal officials, local school 
officials, and local project staft. The table demonstrates the divergence of needs of the 
ihree groups. 

It is important that tliose designing evaluation systems for federally-supported local 
projects recognize these conflicts, and to the extent possible, design systems to meet the 
needs of the various groups. Given the limited resources that have been available for 
evaluation (the median Title VH project evaluation budget in 1989-90 was $3,500), there 
has been a natural tendency on the part of those performing evaluations to focus on the 
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legally-mandated needs of the federal sponsor. However, if the needs of local auditrxes 
for the evaluation are ignored, the data which are collected may not validly reflect actual 
project activities and outcomes (because of low motivation for data collection), and 
important opportunities for project improvement through effective process evaluation 
may be lost. 
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TABLE 2 



Importance Of Various Evaluation Purposes For Different Audiences 



Evaluation Purpose 


Federal Officials 


Local School 
Officials 


Local Project Staff 


Federal 

1. Compliance 
monitoring 


Very important 


Moderately 
important 


Not very important 


2. Grant funding 
evaluation 


Very important 


Not at all important 


Not at all important 


3. Program 
effectiveness 
assessment 


Very important 


Not at all important 


Not at all important 


4. Meta-analytic 
research 


Very important 


Not at all important 


Not very important 


5. Exemplary 
project 
identification 


Moderately 
important 


Not at all important 


Not at all important 


Local 

6. Project 
effectiveness 
assessment 


Not very important 


Very important 


Moderately 
important 


7. Project 
improvement 


Not very important 


Moderately 
important 


Very important 


Federal and Local 

8. Technical 
monitoring 


Very important 


Not very important 


Moderately 
important 


9. Project and 

program 
promotion 


Moderately 
important 


Moderately 
important 


Very important 
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